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(57) Abstract 

In a system in which database images are repre- 
sented by regions having specified feature attributes and 
spatial location attributes, image queries can be directed 
to region feature similarly and region spatial location sim- 
ilarity in combination. If desired, the relative spatial ar- 
rangement of regions can also be taken into account. 
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METHOD AND SYSTEM FOR IMAGE RETRIEVAL 

Technical Field 

This invention relates to computerized image 
retrieval - and, more specifically, to retrieval based on 
image database querying. 

5 Background of the Invention 

With advances in computer hardware technology, 
it has become possible to store, manipulate and transmit 
large numbers of images. When represented in computer- 
tractable form, the images can be included in an image 
10 database. 

Systems have been developed, typically in the 
form of computer software, for image database management 
and image retrieval. For example, as disclosed in 
U.S. Patent 5,493,677, issued February 20, 1996 to Balogh 

15 et al . , images can be retrieved by searching text 

associated with the images for a match with a query. 

Systems have also been developed which use 
image content descriptors, for querying by image content. 
Such a system is disclosed in U.S. Patent 5,579,471, 

20 issued November 26, 1996 to Barber et al . and in the 

paper by W. Niblack et al . , "The QBIC Project: Querying 
Images by Content Using Color, Texture, and Shape", in 
Storage and Retrieval for Image and Video Databases , 
Wayne Niblack, Editor, Proc . SPIE 1908, pp. 173-187 

25 (1993) . 

Summary of the Invention 

We have recognized that, for greater accuracy 
in retrieving images from an image database, querying 
based on image content can be combined with querying 
30 based on spatial location. Thus, in a system in which 
each image is represented by a plurality of regions 
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having feature attributes and spatial location 
attributes, queries can be directed to region feature 
similarity and region spatial location similarity in 
combination. If desired, the relative spatial 
5 arrangement of regions can also be taken into account. 

Brief Description of the Drawing 

Fig. 1 is an example of an image with regions 
for inclusion in a database. 

Fig. 2 is a tabular display of a representation 

10 of the regions of Fig. 1. 

Fig. 3 is an example of an image with regions 

for database querying. 

Fig. 4 is a tabular display of a representation 

of the regions of Fig. 3. 
15 pig. 5 is a flow diagram of database query 

processing for discriminating based on region feature and 

region absolute spatial location. 

Fig. 6 is a flow diagram of database query 

processing for discriminating based on the relative 
20 location of regions. 

Detailed Descriot-.inn of Preferred Embodiments 

The following description is primarily in terms 
of method steps for execution by a suitable processor 
under program control . The program may originate as 
25 • software, or, for greater efficiency, it may be embodied 
at least in part in dedicated firmware or hardware. 

A prototype system embodying features as 
described has been formulated in the JAVA language. The 
system can operate on suitable hardware such as a SUN 
30 Workstation, a Silicon Graphics Workstation, or a PC with 
a Pentium processor, for example. 

Conveniently, an image database to be queried 
has tabular form, with each record or table entry 
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representing a region of an image. A record includes an 
image identifier, a region identifier, a region attribute 
and, for geometric characterization, the x- and y- 
coordinates of the centroid of the region, the width and 
5 height of the region, and the area of the region. The 

table may be generated by manual keyboard entry based on 
visual inspection of images. Alternatively, if a 
.suitable pattern recognition system is available, table 
generation may be automated. 
10 To J illustrate database entries, the image 10 

shown in Fig. 1 and having been given the identifier "T" 
can be represented by the table entries shown in Fig. 2. 
Included are: 

region 100 (t 0 , stretching across the bottom of 
15 the image, below a broken line) ; 

region 101 (t 1# bounded by a rectangle drawn 
with broken lines) ; 

region 102 (t 2 , bounded by a rectangle drawn 
with chain-dotted lines) , 
20 region 103 (t 3 , bounded by a rectangle drawn . 

with broken lines) / and 

region 104 (t 4 , stretching across the top of 
the image, above a broken line) . 

The x, y-coordinates, the width w, and the 
25 height h of each region are given in percent of the 
respective maximal values. The values x, y, w and h 
define a "bounding rectangle" for each region, so that 
the area of a region is less than or equal to w times h. 
As illustrated, regions may overlap, and their union need 
3 0 not cover the image. 

The attribute f may simply represent color, for 
example, with color being represented by known means, 
e«g., b y a color histogram or by color sets. Other 
simple attributes which may be used include texture and 
35 shape, and such attributes may be combined into more 
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complex attributes. 

A search query is expressed correspondingly. 
For example, for the search pattern shown in Fig. 3, 
a query region table may be formed as shown in Fig. 4. 
5 For a database and a query, e.g. with entries 

as illustrated by Figs. 2 and 4, respectively, Fig. 5 
illustrates query processing for finding database entries 
based on the query. The general aim is to find images 
that contain arrangements of regions similar to those in 
10 the query. 

According to Fig. 5, starting at "a" , for each 
region in the query, the database regions are searched 
for a feature match (step 51) and a spatial match 
(step 52) . For spatial matching, this involves using a 
15 suitable metric for comparing the spatial information 

such as x, y, h, w and area of the query region with the 
corresponding information for the database regions . 
Suitable metrics include Euclidean distance and other 
Minkowski distances, and quadratic metrics whose 
20 definition involves a square matrix which expresses the 
relative similarity between the components of a vector. 
A metric can also include weights which may be different 
for each of the geometric parameters. 

Similar metrics can be used for feature 
25 matching (fj . For example, if color histogram 

information is included in terms of components "red", 
"green" and "blue 1 ', a 3-component Euclidean metric can be 
used. Analogously, this applies when such information is 
included in terms of components "hue", "saturation" and 

30 " intensity" . 

For efficiency, as shown in. Fig. 5, thresholds 
are applied to the computed feature and spatial 
distances. Thus, if a distance exceeds the threshold, 
the database region is not included for further 

35 consideration. Instead of, or in addition to using 
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separate thresholds for spatial and feature similarity as 
shown in Fig. 5, thresholding can be applied also to the 
combined region distance or score, i.e. before saving a 
region match in step 53. Distances may be combined by 
simple addition, or by suitable weighting followed by 
addition, for example. 

If multiple processors are available, "k-loop" 
feature similarity processing analogous to step 51 and 
spatial similarity processing analogous to step 52 may be 
carried out in parallel instead of pair-wise sequentially 
as illustrated in Fig. 5. Parallel processing then 
yields two sets of regions, namely (i) those which meet 
feature similarity regardless of spatial similarity, and 
(ii) those which meet spatial similarity regardless of 
15 feature similarity. Thus, to obtain the desired set of 
regions which meet both types of features, a "join" 
operation will be required. After joining, a final 
thresholding operation can be performed. Advantageously, 
multiple processors may also be used for parallel 
20 processing within steps 52 and 53. 

Image matches are obtained as a result of the 
"join" operation in step 54, producing all those database 
images which meet each one of the region requirements of 
the query. A query may result in an image being saved in 
25 step 54 more than once, namely for different combinations 
of its regions which satisfy the query. Such 
multiplicity may be helpful to a user of the system; 
otherwise, duplicates can be deleted by a simple one-pass 
search of the saved images . 
30 If the relative spatial location or arrangement 

of regions is not important to a user, the computation 
may terminate at this point ((3) , though preferably after 
the saved images are sorted by score. 

For discriminating further based on relative 
35 spatial location of regions, a process can be used as 
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illustrated by Fig. 6, using so-called 2-D strings. 
Generation of 2-D strings at this point, i.e. after 
similarity processing, may be termed "query- time 2-D 
string generation" . 

For a query image, a 2-D string includes the x- 
coordinates of the centroids of the regions, arranged as 
an increasing sequence, followed by the y- coordinates of 
the centroids, also arranged as an increasing sequence. 
For a database image, correspondingly, the coordinates of 
those regions are used which were matched against the 

query image regions . 

The 2-D string of the query" image is formed in 
step 61, and, in step 62, this string is matched against 
the 2-D strings from each of the saved images. In 
15 step 63, only in case of a match, the database image is 
saved, so that only those images are ultimately sorted 
and produced in step 64 which have a 2-D string which 
matches the 2-D string of the query image. 

Instead of or in addition to 2-D strings 
including x- and y-coordinates of centroids as described, 
2-D strings can be produced after rotation of the 
coordinate system, e.g. by 45°. Such 2-D strings are 
defined analogously, using coordinates x' and y' of the 
centroids in the rotated coordinate system. 



20 
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CLAIMS : 



1 l. A method for retrieving image representations 

2 from an image database, comprising: 

3 for each of a plurality of regions of a query 

4 image, searching the database for image regions which 

5 match the query region with respect to at least one 

6 feature similarity and at least one spatial similarity; 

7 and 

8 joining the matched image regions. 

1 2, The method according to claim 1, wherein, in 

2 searching the database, feature similarity and spatial 

3 similarity of the database image regions are ascertained 

4 pair-wise sequentially. 

1 3. The method according to claim 1, wherein, in 

2 searching the database, feature similarity and spatial 

3 similarity are ascertained using separate processes in 

4 parallel . 

1 4. The method according to claim 1, further 

2 comprising saving those database images which include the 

3 joined image regions. 

1 5. The method according to claim 4, further 

2 comprising sorting the saved images. 

1 6. The method according to claim 4, further 

2 comprising discriminating among the saved images based on 

3 relative location of the joined regions. 

1 - 7. The method according to claim 6, wherein 

2 discriminating among the saved images comprises comparing 

3 2 -D strings . 
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1 8. The method according to claim 7, wherein the 2- 

2 D strings comprise x- and y- coordinates of region 

3 centroids . 

1 9. The method according to claim 7, wherein the 2- 

2 D strings comprise x' - and y' -coordinates of region 

3 centroids, in a rotated coordinate system. 

1 10. The method according to claim 6, further 

2 comprising sorting the discriminated images. 

1 ll. The method according to claim 4, further 

2 comprising deleting duplicates from the saved images. 

1 12. The method according to claim 1, wherein the 

2 feature similarity comprises color similarity. 

1 13. The method according to claim' 1, wherein the 

2 feature similarity comprises texture similarity. 

1 14. The method according to claim 1, wherein the 

2 feature similarity comprises shape similarity. 

1 15. A system for retrieving image representations 

2 from an image database, comprising: 

3 means for searching the database, for each of a 

4 plurality of regions of a query image, for image regions 

5 which match the query region with respect to at least one 

6 feature similarity, and at least one spatial similarity; 

7 and 

8 means for joining the matched image regions. 

1 16.. The system according to claim 15, wherein the 

2 means for searching the database comprises means for 
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3 ascertaining feature similarity and spatial similarity of 

4 the database image regions pair-wise sequentially. 

1 17. The system according to claim 16, wherein the 

2 means for searching the database comprises means for 

3 ascertaining feature similarity and spatial similarity in 

4 parallel . 

1 18. The system according to claim 15, further 

2 comprising means for saving those database images which 

3 include the joined image regions. 

1 19. The system according to claim 18, further 

2 comprising means for sorting the saved images. 

1 .20. The system according to claim 18, further 

2 comprising means for discriminating among the saved 

3 images based on relative location of the joined regions. 

1 21. The system according to claim 20, wherein, the 

2 means for discriminating among the saved images comprises 

3 means for comparing 2-D strings. 

1 22. The system according to claim 21, wherein the 

2 2-D strings comprise x- and y- coordinates of region 

3 centroids. 

1 23. The system according to claim 21, wherein the 

2 2-D strings comprise x' - and y' -coordinates of region 

3 centroids, in a rotated coordinate system. 

1 24. The system according to claim 20, further 

.2 comprising means for sorting the discriminated images. 

1 25. The system according to claim 18, further 
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2 comprising means for deleting duplicates from the saved 

3 images . 

1 26. The system according to claim 15, wherein the 

2 feature similarity comprises color similarity. 

1 27. The system according to claim 15, wherein the 

2 feature similarity comprises texture similarity. 



1 28. The system according to claim 15, wherein the 

2 feature similarity comprises shape similarity. 
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